Central driver implementation by ntny · Pull Request #13171 · kubeflow/pipelines

ntny · 2026-03-29T20:29:00Z

backup: save central-driver-poc branch before merge
This branch is a backup of central-driver-poc before merging master.
The merge caused too many conflicts for a clean rebase, so the diffs
were extracted into this backup branch to preserve all work safely.
All discussions and comments remain in the original branch.

POC for #12023

Resolves: #12642

feat(backend): introduce centralized driver via Argo executor plugin and eliminate per-component driver pods

Previously, each driver component was executed as a separate Kubernetes pod,
leading to significant overhead in pod scheduling, resource consumption,
and control plane load, especially for pipelines with many small components.

This change replaces the pod-per-driver model with a centralized driver invoked via an Argo Workflow executor plugin using a sidecar-based execution model.

A new driver service is introduced and exposed via an HTTP API, allowing Argo to delegate driver execution without spawning dedicated pods.

Key changes:

Remove pod-per-driver execution model
Introduce centralized driver implemented as an Argo executor plugin
Add driver-plugin sidecar configuration via Kubernetes ConfigMaps
Add new driver implementation (backend/src/driver) with HTTP API
Add required Kubernetes manifests, including RBAC roles and bindings, for the plugin
Integrate plugin configuration into all kustomize deployment variants
Update CI, TLS handling, and debugging scripts to support plugin-based execution
Improve logging with context-aware driver logger
Persist driver logs to S3 and expose them in the UI

Benefits:

Eliminates driver pod churn and reduces scheduling overhead
Reduces pressure on Kubernetes control plane (API server, scheduler, etcd)
Improves execution latency
Enables more efficient and scalable pipeline execution
Provides better observability: driver logs are now available in the UI and persistently stored in S3

This change introduces a new execution model for driver components and lays the groundwork for further optimizations in pipeline runtime.

Description of your changes:

Checklist:

You have signed off your commits
The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

Copilot

Pull request overview

This PR is a backup branch for the “central driver POC” work, transitioning KFP v2 driver execution from per-component driver pods to an Argo Workflow executor plugin + sidecar model, plus associated manifests, RBAC, CI, tests, and UI support for persisted driver logs.

Changes:

Update Argo-compiled workflow “golden” YAMLs to use plugin: driver-plugin templates and add hardened security contexts.
Add/patch Kubernetes manifests (kustomize overlays, RBAC, workflow-controller settings) to enable Argo executor plugins and configure the driver plugin sidecar via ConfigMaps.
Extend backend + frontend to persist driver logs (via MLMD custom properties) and surface them in the UI; adjust tests and CI scripts accordingly.

Reviewed changes

Copilot reviewed 124 out of 209 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
test_data/compiled-workflows/mixed_parameters.yaml	Switch driver execution from container driver pod to `driver-plugin` and add securityContext hardening in golden YAML.
test_data/compiled-workflows/metrics_visualization_v2.yaml	Same as above for metrics visualization golden YAML.
test_data/compiled-workflows/lightweight_python_functions_with_outputs.yaml	Same as above for lightweight python functions (outputs) golden YAML.
test_data/compiled-workflows/lightweight_python_functions_pipeline.yaml	Same as above for lightweight python functions pipeline golden YAML.
test_data/compiled-workflows/iris_pipeline_compiled.yaml	Same as above for iris pipeline golden YAML.
test_data/compiled-workflows/if_else_with_oneof_artifacts.yaml	Same as above for if/else (oneof artifacts) golden YAML.
test_data/compiled-workflows/if_elif_else_with_oneof_parameters.yaml	Same as above for if/elif/else (oneof parameters) golden YAML.
test_data/compiled-workflows/identity.yaml	Same as above for identity golden YAML.
test_data/compiled-workflows/hello_world.yaml	Same as above for hello world golden YAML.
test_data/compiled-workflows/flip_coin.yaml	Same as above for flip coin golden YAML.
test_data/compiled-workflows/embedded_artifact.yaml	Same as above for embedded artifact golden YAML.
test_data/compiled-workflows/dict_input.yaml	Same as above for dict input golden YAML.
test_data/compiled-workflows/container_io.yaml	Same as above for container IO golden YAML.
test_data/compiled-workflows/concat_message.yaml	Same as above for concat message golden YAML.
test_data/compiled-workflows/component_with_metadata_fields.yaml	Same as above for component metadata fields golden YAML.
test_data/compiled-workflows/artifact_crust.yaml	Same as above for artifact crust golden YAML.
test_data/compiled-workflows/artifact_cache.yaml	Same as above for artifact cache golden YAML.
test_data/compiled-workflows/arguments_parameters.yaml	Same as above for arguments parameters golden YAML.
test_data/compiled-workflows/add_numbers.yaml	Same as above for add numbers golden YAML.
manifests/kustomize/third-party/argo/installs/namespace/workflow-controller-argo-role-patch.json	Patch Workflow Controller role to add WorkflowTaskSet status permissions for agent/plugin flow.
manifests/kustomize/third-party/argo/base/workflow-controller-deployment-patch.yaml	Enable Argo executor plugins via env vars.
manifests/kustomize/third-party/argo/base/workflow-controller-argo-taskset-clusterrole-.yaml	Add cluster-scoped RBAC for WorkflowTaskSet access needed by Argo agent/plugin.
manifests/kustomize/third-party/argo/base/kustomization.yaml	Include the new WorkflowTaskSet RBAC resource.
manifests/kustomize/third-party/application/application-controller-deployment.yaml	Increase application-controller memory request.
manifests/kustomize/env/dev/kustomization.yaml	Pin `kfp-driver` image tag and add dev overlay patch for driver plugin ConfigMap.
manifests/kustomize/env/dev/driver-plugin-cm-path.yaml	Dev overlay driver plugin sidecar ConfigMap (image + env + resources + securityContext).
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/ml-pipeline-ui-deployment.yaml	Update TLS secret reference to Argo agent CA secret name.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/ml-pipeline-scheduledworkflow-deployment.yaml	Same TLS secret rename.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/ml-pipeline-persistenceagent-deployment.yaml	Same TLS secret rename.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/ml-pipeline-driver-plugin-cm.yaml	TLS overlay patch for driver plugin sidecar ConfigMap (+ cert mount).
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/ml-pipeline-apiserver-deployment.yaml	Update CABUNDLE secret name + volume secret for TLS.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/metadata-writer-deployment.yaml	Same TLS secret rename.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/metadata-grpc-deployment.yaml	Same TLS secret rename.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/metadata-envoy-deployment.yaml	Same TLS secret rename.
manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/kustomization.yaml	Apply driver plugin ConfigMap patch in TLS overlay.
manifests/kustomize/env/cert-manager/base-tls-certs/kfp-api-cert.yaml	Rename cert secret to match Argo agent’s required secret name.
manifests/kustomize/base/pipeline/pipeline-runner-sa.yaml	Add explicit SA token Secret for `pipeline-runner`.
manifests/kustomize/base/pipeline/pipeline-runner-role.yaml	Expand RBAC for WorkflowTaskSet resources and add executor-plugin service account + roles/bindings.
manifests/kustomize/base/pipeline/ml-pipeline-driver-plugin-cm.yaml	Base driver plugin sidecar ConfigMap (ExecutorPlugin type).
manifests/kustomize/base/pipeline/kustomization.yaml	Add driver plugin ConfigMap to base resources.
manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/sync.py	Add Kubernetes client usage to provision executor-plugin SA and RBAC per namespace.
manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/pipelines-profile-controller-admin.yaml	Add admin SA + ClusterRole/Binding for profile controller to manage SA/RBAC.
manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/kustomization.yaml	Include the new admin RBAC resource.
manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/deployment.yaml	Run profile controller with admin SA; install Python deps at runtime.
frontend/src/hooks/queryKeys.ts	Add React Query key for driver/system logs.
frontend/src/components/tabs/RuntimeNodeDetailsV2.tsx	Add “System Logs” tab and fetch driver logs artifact via API.
backend/test/testutil/kubernetes_utils.go	Formatting cleanup.
backend/test/end2end/utils/e2e_utils.go	Capture workflow-controller logs for failed runs and handle agent pod logs.
backend/test/end2end/e2e_suite_test.go	Formatting cleanup.
backend/test/compiler/utils/workflow_utils.go	Ensure cache settings updates also apply to the driver plugin args.
backend/src/v2/metadata/client.go	Persist `driver_logs_uri` (+ store session info) in execution custom properties.
backend/src/v2/driver/root_dag.go	Use context-aware logger and return pipeline from RootDAG.
backend/src/v2/driver/k8s_test.go	Ensure tests provide a logger via context.
backend/src/v2/driver/k8s.go	Replace glog with context logger; require logger in context for several paths.
backend/src/v2/driver/driver_test.go	Ensure tests provide a logger via context.
backend/src/v2/driver/driver.go	Change output URI prefix naming used for task-root generation.
backend/src/v2/driver/dag.go	Avoid re-fetching pipeline in DAG driver; use context logger.
backend/src/v2/driver/container.go	Pass output prefix; write DriverLogURI custom property; use context logger.
backend/src/v2/compiler/argocompiler/plugin.go	Add helper to build Argo executor plugin template JSON.
backend/src/v2/compiler/argocompiler/dag.go	Switch DAG driver template to `Plugin` with args map; adjust call sites for error handling.
backend/src/v2/compiler/argocompiler/container_test.go	Skip env var test that no longer applies under plugin-based execution.
backend/src/v2/compiler/argocompiler/container.go	Switch container driver template from Container to Plugin, including JSONPath outputs.
backend/src/v2/compiler/argocompiler/argo.go	Disable Istio sidecar injection on workflow pods; remove driver image/command from compiler state.
backend/src/v2/cmd/driver/execution_paths.go	Remove old v2 driver execution paths file (moved).
backend/src/driver/main_test.go	Update test expectations for renamed symbols and parsing function name.
backend/src/driver/main.go	New driver plugin HTTP server entrypoint and shared helpers.
backend/src/driver/execution_paths.go	Re-add ExecutionPaths struct under new driver plugin package.
backend/src/driver/api/response.go	Add HTTP response DTOs for driver server/plugin execution.
backend/src/driver/api/request.go	Add HTTP request DTOs for driver server/plugin execution.
backend/src/common/util/context_logger.go	Introduce context logger (logrus) writing to stdout + file.
backend/Dockerfile.driver	Build the new driver plugin binary from `backend/src/driver`.
.github/workflows/legacy-v2-api-integration-tests.yml	Update CA secret name used by CI.
.github/workflows/e2e-test.yml	Update CA secret name used by CI.
.github/workflows/api-server-tests.yml	Update CA secret name used by CI.
.github/resources/scripts/kfp-readiness/wait_for_pods.py	Exclude `*-agent` pods from readiness filter.
.github/resources/scripts/collect-logs.sh	Add agent container log collection and describe workflows.
.github/resources/manifests/standalone/tls-enabled/kustomization.yaml	Patch driver plugin ConfigMap in TLS-enabled GH manifests.
.github/resources/manifests/standalone/tls-enabled/driver-plugin-cm-path.yaml	Provide driver plugin sidecar ConfigMap for TLS-enabled GH manifests.
.github/resources/manifests/standalone/proxy/kustomization.yaml	Patch driver plugin ConfigMap in proxy GH manifests.
.github/resources/manifests/standalone/default/kustomization.yaml	Patch driver plugin ConfigMap in default GH manifests.
.github/resources/manifests/standalone/cache-disabled/kustomization.yaml	Patch driver plugin ConfigMap in cache-disabled GH manifests.
.github/resources/manifests/standalone/cache-disabled-proxy/kustomization.yaml	Patch driver plugin ConfigMap in cache-disabled-proxy GH manifests.
.github/resources/manifests/multiuser/default/kustomization.yaml	Patch driver plugin ConfigMap in multiuser GH manifests.
.github/resources/manifests/multiuser/cache-disabled/kustomization.yaml	Patch driver plugin ConfigMap in multiuser cache-disabled GH manifests.
.github/resources/manifests/multiuser/artifact-proxy/kustomization.yaml	Patch driver plugin ConfigMap in artifact-proxy GH manifests.
.github/resources/manifests/kubernetes-native/default/kustomization.yaml	Patch driver plugin ConfigMap in kubernetes-native GH manifests.
.github/resources/manifests/base/driver-plugin-cm-path.yaml	Base GH manifest overlay for driver plugin sidecar ConfigMap.

Copilot · 2026-03-29T20:31:38Z

+        command:
+          - sh
+          - -c
+          - |
+            python3 -m venv /tmp/venv && \
+            . /tmp/venv/bin/activate && \
+            pip install --no-cache-dir kubernetes boto3 && \
+            python /hooks/sync.py


Installing dependencies at container startup is operationally risky:

Requires outbound network access to PyPI (often blocked in clusters).

Adds cold-start latency and makes behavior non-reproducible.

Can fail if python3, venv, or pip are not present in the base image.

Fix (required): bake kubernetes + boto3 into the image used for the profile controller (or switch to an image that already contains them), and run python /hooks/sync.py directly.

Hopefully addressed by ntny#2

The metacontroller and pipelines profile controller is different. PPC only provides a json to metacontroller and metacontroller handles the rest.

see see https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/third-party/metacontroller/base/cluster-role.yaml

and https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/decorator-controller.yaml

Copilot · 2026-03-29T20:31:38Z

 		return execution, err
 	}
 	ecfg.TaskName = opts.TaskName
+	ecfg.Namespace = opts.Namespace


ExecutionConfig (in backend/src/v2/metadata/client.go) does not show a Namespace field in the diff, so ecfg.Namespace = opts.Namespace is very likely a compile-time error.

Fix (required): either:

add a Namespace field to metadata.ExecutionConfig and plumb it through (if needed), or

remove this assignment if namespace is already available via opts.Namespace / pipeline context.

Suggested change

ecfg.Namespace = opts.Namespace

ntny · 2026-03-29T21:51:43Z

/retest

ntny · 2026-03-29T22:18:36Z

/retest

ntny · 2026-03-29T22:46:34Z

/retest

ntny · 2026-03-30T15:50:08Z

/retest

juliusvonkohout

Hello, I am really concerned about the sync.py changes. Why do you want to create new serviceaccounts? Pipelines usually run as default-editor in Kubeflow. Usually such roles should be aggregated to kubeflow-edit. Adding attidional service accounts i really want to avoid. That will probably conflict with the average multi-user Kubeflow and create technical debt.

juliusvonkohout · 2026-03-31T18:49:24Z

i saw this patch in multiple places. Can we consolidate ?

juliusvonkohout

I added a few comments. The PipelineProfileController+Metacontroller architecture is different than what you expect your code does not match it.

See https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/decorator-controller.yaml and https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/third-party/metacontroller/base/cluster-role.yaml. The pipelines profile controller just provides the input for metacontroller.

You also patch the permissions for "workflowtasksets/status" in many files which adds a lot of code duplication.

The PSS restricted stuff looks good.

ntny · 2026-04-02T07:10:57Z

/retest

ntny · 2026-04-02T08:13:28Z

/retest

google-oss-prow · 2026-05-30T14:24:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hbelmiro, humairak for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

juliusvonkohout · 2026-06-02T10:44:50Z

please rebase to master

…and eliminate per-component driver pods Previously, each driver component was executed as a separate Kubernetes pod, leading to significant overhead in pod scheduling, resource consumption, and control plane load, especially for pipelines with many small components. This change replaces the pod-per-driver model with a centralized driver invoked via an Argo Workflow executor plugin using a sidecar-based execution model. A new driver service is introduced and exposed via an HTTP API, allowing Argo to delegate driver execution without spawning dedicated pods. Key changes: - Remove pod-per-driver execution model - Introduce centralized driver implemented as an Argo executor plugin - Add driver-plugin sidecar configuration via Kubernetes ConfigMaps - Add new driver implementation (backend/src/driver) with HTTP API - Add required Kubernetes manifests, including RBAC roles and bindings, for the plugin - Integrate plugin configuration into all kustomize deployment variants - Update CI, TLS handling, and debugging scripts to support plugin-based execution - Improve logging with context-aware driver logger - Persist driver logs to S3 and expose them in the UI Benefits: - Eliminates driver pod churn and reduces scheduling overhead - Reduces pressure on Kubernetes control plane (API server, scheduler, etcd) - Improves execution latency - Enables more efficient and scalable pipeline execution - Provides better observability: driver logs are now available in the UI and persistently stored in S3 This change introduces a new execution model for driver components and lays the groundwork for further optimizations in pipeline runtime. Signed-off-by: arpechenin <arpechenin@avito.ru>

Signed-off-by: arpechenin <arpechenin@avito.ru>

- fix rbac granting - drop redundant patch Signed-off-by: arpechenin <arpechenin@avito.ru>

- rewrite tests - move updated logic from the kubeflow@f8ee99c#diff-937c57a877a615c510d37e38309f0b159c85595d2dfa908d9e9ef05c227064da Signed-off-by: arpechenin <arpechenin@avito.ru>

Replaced the namespace-scoped Role with a ClusterRole to avoid creating the same role in every namespace. Signed-off-by: arpechenin <arpechenin@avito.ru>

Signed-off-by: arpechenin <arpechenin@avito.ru>

- added explanation for list pods RBAC - enhanced post-e2e test debugging - tweaked driver resources - added comments clarifying why extra RBAC is required - fixed images to match master - removed secret duplication and hardcoded namespace - update compiler golden files Signed-off-by: arpechenin <arpechenin@avito.ru>

Driver templates now run as Argo executor plugins instead of workflow container templates, so run-specific env injection no longer applies to them. Store driver runtime values under driver-plugin args and initialize task plugins from those args, while keeping container env injection for launcher templates unchanged. Signed-off-by: arpechenin <arpechenin@avito.ru>

ntny · 2026-06-13T21:07:25Z

/retest

ntny · 2026-06-13T21:38:24Z

Well, all tests are finally passing

On Monday I'll post an update with the current status and a summary of what changed during the rebase.

ntny · 2026-06-15T09:53:34Z

Hi @mprahl, could you please take a look at the MLflow runtime config changes for the driver plugin?

Since the driver is now an Argo executor plugin, its env vars and pod annotations are only available for global plugin configuration. Because of that, I no longer pass KFP_MLFLOW_CONFIG to the driver through env vars.

Previously the flow was:

add the driver role annotation to the pod in the compiler
in UpsertRuntimeEnvVars, add KFP_MLFLOW_CONFIG to the driver env vars

Now I detect the driver plugin via tmpl.Plugin != nil instead of the driver's pod annotation role. If the driver role is requested, I update driver-plugin.args.runtime_args instead of env vars and put the runtime config

The driver plugin then parses runtime_args here

I also added MLflow RBAC (mlflow.kubeflow.org/experiments get/list/create/update) for the separate driver plugin service account. Without this, the API server can create the root run successfully, but task-level nested runs from the driver plugin get 403, because MLflow Kubernetes auth checks the bearer token from the pod service account.

Copilot AI review requested due to automatic review settings March 29, 2026 20:29

google-oss-prow Bot requested review from droctothorpe, gmfrasca, juliusvonkohout, mprahl and nsingla March 29, 2026 20:29

google-oss-prow Bot added the size/XXL label Mar 29, 2026

Copilot AI reviewed Mar 29, 2026

View reviewed changes

ntny mentioned this pull request Mar 29, 2026

Central driver POC #12269

Closed

ntny force-pushed the central-driver-poc-backup branch from 1ad5ac7 to de96fee Compare March 29, 2026 21:13

droctothorpe force-pushed the central-driver-poc-backup branch from de96fee to 0a4db0b Compare March 30, 2026 11:04

ntny changed the title ~~Central driver POC backup~~ Central driver implementation Mar 30, 2026

ntny force-pushed the central-driver-poc-backup branch from da711da to 93b2f1a Compare March 31, 2026 08:03

juliusvonkohout requested changes Mar 31, 2026

View reviewed changes

Comment thread manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/sync.py

google-oss-prow Bot assigned juliusvonkohout Mar 31, 2026

juliusvonkohout reviewed Mar 31, 2026

View reviewed changes

Comment thread manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/deployment.yaml Outdated

juliusvonkohout reviewed Mar 31, 2026

View reviewed changes

Comment thread ...ase/installs/multi-user/pipelines-profile-controller/pipelines-profile-controller-admin.yaml Outdated

juliusvonkohout reviewed Mar 31, 2026

View reviewed changes

Comment thread manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/kustomization.yaml Outdated

juliusvonkohout requested changes Mar 31, 2026

View reviewed changes

ntny force-pushed the central-driver-poc-backup branch 3 times, most recently from ca40a3c to 23456c9 Compare April 2, 2026 06:33

zazulam requested a review from juliusvonkohout April 3, 2026 21:14

ntny force-pushed the central-driver-poc-backup branch from e7e61b8 to e85dac2 Compare May 30, 2026 16:52

ntny force-pushed the central-driver-poc-backup branch from e85dac2 to 6ca0446 Compare June 11, 2026 20:40

arpechenin added 10 commits June 11, 2026 23:41

- fix manifest naming, logs error message, rbac rules

f98d7a9

Signed-off-by: arpechenin <arpechenin@avito.ru>

- restrict access to secrets to the profile-controller component.

58870dc

Signed-off-by: arpechenin <arpechenin@avito.ru>

- merge tests

7ac345c

Signed-off-by: arpechenin <arpechenin@avito.ru>

- remove k8s python dep from the sync.py

ec44c2c

- fix rbac granting - drop redundant patch Signed-off-by: arpechenin <arpechenin@avito.ru>

- resolve conflicts

727bb23

- rewrite tests - move updated logic from the kubeflow@f8ee99c#diff-937c57a877a615c510d37e38309f0b159c85595d2dfa908d9e9ef05c227064da Signed-off-by: arpechenin <arpechenin@avito.ru>

Merged all RBAC roles for the executor plugin into a single one.

1121629

Replaced the namespace-scoped Role with a ClusterRole to avoid creating the same role in every namespace. Signed-off-by: arpechenin <arpechenin@avito.ru>

fix err swallowing

8ff6402

Signed-off-by: arpechenin <arpechenin@avito.ru>

fix parameter passing

6acc282

Signed-off-by: arpechenin <arpechenin@avito.ru>

ntny force-pushed the central-driver-poc-backup branch 11 times, most recently from 0a89003 to 499c666 Compare June 13, 2026 15:15

ntny force-pushed the central-driver-poc-backup branch from 499c666 to 5dda2a6 Compare June 13, 2026 15:56

Conversation

ntny commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

droctothorpe Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ntny commented Mar 29, 2026

Uh oh!

ntny commented Mar 29, 2026

Uh oh!

ntny commented Mar 29, 2026

Uh oh!

ntny commented Mar 30, 2026

Uh oh!

juliusvonkohout left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

juliusvonkohout Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juliusvonkohout left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ntny commented Apr 2, 2026

Uh oh!

ntny commented Apr 2, 2026

Uh oh!

google-oss-prow Bot commented May 30, 2026

Uh oh!

juliusvonkohout commented Jun 2, 2026

Uh oh!

ntny commented Jun 13, 2026

Uh oh!

ntny commented Jun 13, 2026

Uh oh!

ntny commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ntny commented Mar 29, 2026 •

edited

Loading

juliusvonkohout left a comment •

edited

Loading

juliusvonkohout left a comment •

edited

Loading